Codecov Report

Merging #3508 (b333a8f) into main (483cc31) will decrease coverage by 7.62%.
The diff coverage is 75.78%.

@@            Coverage Diff             @@
##             main    #3508      +/-   ##
==========================================
- Coverage   76.39%   68.78%   -7.62%     
==========================================
  Files         210      212       +2     
  Lines       14421    14496      +75     
==========================================
- Hits        11017     9971    -1046     
- Misses       2757     3946    +1189     
+ Partials      647      579      -68     

The check also looks for the installation of maven (or gradle) Action to verify the integrity of wrapper binaries. I think we need more than one probe to handle these cases. Suggestion: hasMavenWrapperBinary, hasMavenWrapperActionInstalled, hasOtherBinaries.

The check also looks for the installation of maven (or gradle) Action to verify the integrity of wrapper binaries. I think we need more than one probe to handle these cases. Suggestion: hasMavenWrapperBinary, hasMavenWrapperActionInstalled, hasOtherBinaries.

I only see checks for Gradle Wrappers, however, the filtering happens before the files are added to the BinaryArtifactData. As such, the current Scorecard check + evaluation does not filter out wrapper binaries. Is that something we should change and convert to probes?

The check also looks for the installation of maven (or gradle) Action to verify the integrity of wrapper binaries. I think we need more than one probe to handle these cases. Suggestion: hasMavenWrapperBinary, hasMavenWrapperActionInstalled, hasOtherBinaries.

I only see checks for Gradle Wrappers, however, the filtering happens before the files are added to the BinaryArtifactData. As such, the current Scorecard check + evaluation does not filter out wrapper binaries. Is that something we should change and convert to probes?

yes I think that would be useful. I could image end-users wanted to know if there's a gradle wrapper present in the repo or not. I think it's ok to do this in this. @spencerschrock wdut?

I only see checks for Gradle Wrappers, however, the filtering happens before the files are added to the BinaryArtifactData. As such, the current Scorecard check + evaluation does not filter out wrapper binaries. Is that something we should change and convert to probes?

yes I think that would be useful. I could image end-users wanted to know if there's a gradle wrapper present in the repo or not. I think it's ok to do this in this. @spencerschrock wdut?

I think it's fine to do the filtering on the evaluation side, but would need to double check against the Allstar code.

The check also looks for the installation of maven (or gradle) Action to verify the integrity of wrapper binaries. I think we need more than one probe to handle these cases. Suggestion: hasMavenWrapperBinary, hasMavenWrapperActionInstalled, hasOtherBinaries.

With regard to probes, I haven't read the PR, but my proposed breakdown was probably going to have multiple probes (although only one would likely be used by the evaluation code):

• free of binary artifacts
• free of binary artifacts (except for the "approved" maven ones)

I think this is another example where it makes sense to clarify what a finding is? If it's some data point which optionally has a file location, then would each binary artifact have its own finding? (note: this is a little different than the repeated contributor/commit findings). If it's gradle verified it could be a gradle validated finding, otherwise a normal binary finding.

And the policy could later walk both, and choose to either count or ignore the gradle verified ones

Stale pull request message

This pull request is stale because it has been open for 10 days with no activity

With regard to probes, I haven't read the PR, but my proposed breakdown was probably going to have multiple probes (although only one would likely be used by the evaluation code):

• free of binary artifacts
• free of binary artifacts (except for the "approved" maven ones)

I believe this is the outcome of our discussion today. @laurentsimon your comment about different logic for each finding, I don't think that's true so long as the policy picks one.

e.g. if you have 3 binaries: foo, bar, and gradle-wrapper.jar (verified)

The first probe would complain about all 3.
the second probe would only complain about foo and bar.

Scorecard evaluation code would use the second probe. The first probe would be dangling/uncalled, until someone writes a policy to use it.

With regard to probes, I haven't read the PR, but my proposed breakdown was probably going to have multiple probes (although only one would likely be used by the evaluation code):

• free of binary artifacts
• free of binary artifacts (except for the "approved" maven ones)

I believe this is the outcome of our discussion today. @laurentsimon your comment about different logic for each finding, I don't think that's true so long as the policy picks one.

I think I was wrong too. So long as there's one finding per binary, it should work. So the main benefit of having different probes would be to accommodate different remediations for each "type" of verification. Today I don't know if we need this flexibility. But if we want to not be blocked in the future, that could be beneficial to do it now. I don't see a strong argument for trying to pack multiple verification types into a single "unverifiedProbe". What is the main motivation?

e.g. if you have 3 binaries: foo, bar, and gradle-wrapper.jar (verified)

The first probe would complain about all 3. the second probe would only complain about foo and bar.

I thought we agreed the were mutually exclusive? (can re-visit if it's better not to)

Scorecard evaluation code would use the second probe. The first probe would be dangling/uncalled, until someone writes a policy to use it.

I thought we agreed the were mutually exclusive? (can re-visit if it's better not to)

I thought the binary type was mutually exclusive, but the probes may not be. Can discuss offline if needed.

/scdiff generate Binary-Artifacts

Here's a link to the scdiff run